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Kit* The concept of HIS is in naserons ways sore flexible than sight 
be isagined froa a cursory reading of the business-sanagesent 
literature* An HIS does not have to be an expensive, cosputerized 
systes* An econosical, lannal HIS can be created to seet the needs of 
a single school. The csE Forsative Evaluation Kit operates ais an HIS 
to provide a principal vith inforsation on a nev progras. Heavy 
esphasis is given to planning activities that relate directly to 
analysis of progras perforsance and decisionsaking. The CSE kit users 
are directed to plan evaluation activities every tise sajor progras 
cosponents are changed. Pilot-test and field-test results suggest 
that the kit is useful vith a variety of prograss. The kit has the 
capability of saking isportant planning activities a forsal process. 
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Introduction 

References to the use of manafjement information , management information 
systems ■ and systems management began appearing in the educational literature 
about 10 years aqo. Both the vocabulary and the substantive techniques 
behind the vocabulary were borrowed from the world of business, much as the 
efficiency moviiment in education had done in the 1920's. The reasons for 
this recent borrowing were varied — the success of systems approaches 
as used in industry (especially the defense industry), the subsequent adop- 
tion of systems procedures by the U.S. Office of Education and» perhaps more 
fundamentally, the growing faith that increased clarity and rationality 
were needed if better solutions were to be found to contemporary educational 
problems. Historical descriptions of this trenu can be found in Handy and 
Hussain (1969), Banghart (1969), and Yee (1973). 

This paper will deal with one of these systems concepts, namely manage- 
ment information systems (MIS), especially as exemplified by the CSE Formative 
Evaluation KIT. It will show that the concept of MIS is in numerous ways, 
more flexible than might be imagined from a cursory reading of the business- 
management literature: 

1. An MIS does not have to be an expensive, computerized system. An 
economical, manual MIS can be created for tht needs of a single 
school . 

2. The ''SE Formative Evaluation KIT operates as an MIS to provide 
information on a new program to a principal. Heavy emphasis is 
given to planning activities that relate directly to analysis of 
program performance and decision inaking. KIT users are directed 
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to plan evaluation activities every time major program components 
are changed. 

3. Pilot-test and field-test results suggest that the KIT Is useful 
with a variety of programs. The KIT has the capability of making 
Important planning activities a formal process. 

Computerization and MIS 

Many business-management authors stress the automated aspects of MIS. 
For example, Davis (1974) defines an MIS In this way: ". . . an Integrated 
man/machine system fci providing Information to support the operations, 
management and decision-making functions In an organization. The system 
utilizes computer hardware and software, manual procedures, management and 
decision models and a data base." (p. 5) Although some consideration is 
given to "manual procedures," the emphasis here Is clearly on a computerized 
system. A similar connotation Is found In O'Brien (1970): "A useful MIS 
accumulates, processes stores and transmits data to relevant people In 
management." (p. 11) 

The description of MIS by these authors (and most others of similar 
background) basically expounds a common theme — hew to Integrate the Informa 
tlon needs of a manager with a computerized system. Topics such as Informa- 
tion data banks and transaction processing systems are considered at some 
length. At first glance. It may appear that an MIS without a computer Is a 
contradiction In terms. That this Is not true becomes clear only after a 
consideration of both the basic nature of a system and the range of environ- 
ments In which an MIS can operate. 
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Whatever else It 1s» a system essentially Is a set of components work- 
ing toward a common objective. As C. W. Churchman puts It, "Systems are 
made up of sets of components that work together for the overall objective 
of the whole. The systems approach Is simply a way of thinking about these 
total systems and their components." (p. 11) A management-Information 
system Is a set of components that work toward the objective of providing a 
manager with Information. There Is nothing In the basic conception of an 
MIS that requires computerization. Certainly, computerized systems are 
not a disadvantage. But the context In which they are to be used must 
be appropriate. 

A typical computerized MIS exists In a business organization marked by 
large-scale production processes. Inventory control, and marketing demands. 
Computerized MIS do exist In higher education and In local school districts 
vis-a-vis district-wide programs and district-wide needs. But an Individual 
principal Is as much a manager as a superintendent Is; and although the 
principal's organization Is obviously smaller, his needs for management 
information are as real. A small, manual MIS can be as much a system as a 
large-scale computerized MIS. 

How the KIT Operates as a System . The CSE Formative Evaluation KIT 
provides the principal with an MIS appropriate for an Individual school 
program In its development stage. It Is an MIS because all of Its parts aim 
at providing formative evaluation information — as distinct from information 
gathered for needs assessment, program planning, or summative evaluation. 
In the Center for the Study of Evaluation (CSE) model (Klein, Fenstermacher, 
& Alkin, 1971), Implementation Evaluation and Progress Evaluation serve the 



formative function. Formative Evaluation has as Its aim program Improvement 
through decisions to change a new program while It Is going on. 

Implementation Evaluation focuses on the extent of program Installation 
and Progress Evaluation on the extent of the program's performance In meeting 
objectives. Since Implementation begins as soon as the program begins and 
progress of some sort should be occurring the entire length of the program. 
Information Is needed quickly after the program starts and In recurring 
cycles after each change In program direction. Information collected In the 
course of Formative Evaluation should be relevant and timely for the fluid 
events of an evolving program. 

An overview of the 16 Steps of the KIT Is shown In Figure 1. The figure 
shows that the Initial 8 Steps of the KIT are devoted to planning activities. 
The first time they are ffsed (the "first Iteration") the program has not 
started yet. Thus, the first half of the KIT Is used for evaluation planning 
before Instructional staff and students actually begin program processes; 
many of the Steps are used again after some Information Is gathered on the 
performance of the program. 

The second half of the KIT Is devoted to collecting, analyzing, and 
reporting Information that was planned with the first set of steps. They 
also guide the user In recycling to do more planning based upon Initial 
results of the program. 

The first 8 Steps or "planning half" of the Formative KIT, Including Its 
Information outputs. Is detailed In Figure 2. The Information outputs are 
In the form of Worksheets completed by the principal in cooperation with the 
Instructional staff. Although the Worksheets are completed one at-a-tlme. 
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Figure 1 

Overview of the CSE Formative Evaluation KIT 





1. 


Reviewing the Program Plan 




2. 


Constructing a Master Plan 


Steps 




Developing Evaluation Questions 


1 through 8 


4. 


Selecting a Measurement Strategy 




5. 


Estimating Evaluation Resources 


PLANNING 






6. 


Determining Appropriate Measures 


m 


7. 


Setting Evaluation Deadlines 




8. 


Establishing Communication 


• 


9. 


Collecting Data 




10. 


Discovering Unanticipated Outcomes 


Steps 


11. 


Scoring Measures 


9 i:hrough 16 






12. 


Preparing Data for Presentation 


ANALYZING 


13. 


Making Recommendations 


and 


—14. 


Implementing Recommended Changes 


REPORTING 






(15.) 


Deleted 




16. 


Preparing Reports 



ERIC 



5 

7 



Figure 2 

Overview of the Planning Phase of the CSE Formative KIT 
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there Is not a strict linear relationship among them (e.g.. each one 
requiring the absolute completion of the preceding). Nor are they all 
equal In Importance — some are clearly subsidiary to others 'In scope and 
function. 

Worksheets B» C» and F are the keystones of the planning steps: B 
summarizes staff preparatory activities and major student objectives; 
C elicits evaluation questions that should be answered as the program Is 
Installed and In operation; and F requires the setting down of deadlines for 
when Information should be reported regarding the evaluation questions. 
Worksheet A serves a basic need since the user must make a master plan and 
develop questions and can do so only If the program plan has been verified, 
e.g., there Is general agreement on the content and boundaries of the pro- 
gram. Worksheets 0 and E are Important because they requ1:e a considera- 
tion of the constraints operating In the system (money for purchase of 
tests and employment of personnel, time for acquisition or construction of 
Instruments). Planning of this type fosters the setting of realistic 
deadlines. 

The planning approach in these early steps follows what G. B. Davis, 
speaking from a business-management orientation, calls a "top-down" rather 
than a "bottom-up" or "evolutionary" approach to MIS development. Explain- 
ing the former approach, Davis (1974) says, "In order to define the overdll 
system plan, the top-down analysis approach begins by defining the objectives 
of the organization, the kind of business it is In, and the constraints under 
which it operates. The activities or functions are identified. The crucial 
strategic and tactical decisions are then defined and the decisions necessary 
to operate the activities are specified." (p. 408) 



The bottom-MP approach 1$, logically enough, the Inverse of the top- 
down. The order of activities Is "backwards" In that the existing informa- 
tion systems are used as a foundation and are built upon as needed to 
satisfy the decision needs of the manager. The top-down approach used by the 
KIT Is In keeping with the nature of Formative Evaluation. With Formative 
Evaluation, the user Is looking at a new program. It Is critical to Identify 
clearly at the outset the objectives of the program, formulate relevant 
questions on these objectives, and select the proper measures to administer. 
There Is little likelihood that pre*ex1 sting testing systems or other-directed 
Information gathering can provide necessary data on a new program. Further, 
since students may be heavily tested as a part of the regular school routine, 
specifying objectives and carefully selecting measures for them at the out- 
set minimizes possible problems of overtesting or confused or conflicting 
scheduling of tests. 

The Steps devoted to analyzing and reporting are shown In Figure 3. In 
the two right-hand columns following the Step title are entries for Infor- 
mation-gathering processes used In the Step and output yielded by the Step. 
All outputs are program-relevant, e.g.. critical tasks directly Involving 
monitoring of program Implementation and progress. In contrast, outputs for 
planning Steps served to yield planning Information only. 

Although Steps 9 to 16 are all of great Importance. 10, 13. and 14 are 
the critical ones. Step 10 focuses on an often-neglected topic — unintended 
outcomes as revealed by unobtrusive measures. These outcomes serve as an 
Important back-up device for more traditional measurement techniques by try- 
ing to ensure that all outcomes receive examination In assessing program 
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Figure 3 

Overview of the Operational Phase of the CSE Formative KIT 
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performance. Step 13 gives the user a technology for decision making 
a simple group-discussion approach that minimizes complex statistics and 
maximizes involvement of all teachers In the program. The Step Is anal- 
ogous to the component of a business MiS where a decision rule Is used to 
guide the manager In his actions. This Is the point at which decisions are 
made regarding which program components to change and how to change them. 
The third crucial Step, number 14, serves a dual function: It returns 
the user to Step 3 for new planning activities and It provides a checklist 
to determine whether the recommended changes are. In fact, carried out. 
This checklist Is called a Worksheet although It does not serve the same 
purpose as Worksheets A to H. 

The KIT gives the user what Davis calls "Information system support 
for decision making." (p. 319) It provides Information on many aspects of 
the program, so that by the time Step 13 Is reached there Is sufficient 
basis for choosing which components of the program could benefit from change. 

Routlnlzing Formative Evaluation for Many Types of Programs . One of 
the goals of the KIT Is to make previously "hit-or-miss" procedures a part 
of the regular operation of the school. This function Is well summarized 
by the phrase "routlnlzing evaluation" an expression used by sociologist 
Daniel Glaser (1973) In describing his ideal of evaluation procedures in 
crime-and delinquency-prevention programs. Evidence is presented on the 
KIT'S potential of making evaluation procedures routine administrative 
activities. 

While in its development stages, a draft version of the Formative KIT 
was produced and circulated for use and/or review by several southern 
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California elementary schools. Principals were asked to read the material 
In the KIT., use It wherever possible, and compare It to previously 
employed procedures. Participants were then Interviewed about the KIT. 
Table 1 shows responses to several key questions on the material presented 
In the pilot-test version. 

Table 1 

Number of Responses of Pilot-Test Users to 
Selected Questions on the CSE Formative 
Evaluation KIT 





How pertinent was this 
Step to conducting an 
evaluation? 


Were the activities 
required by this 
Step ever done before 
with a new program? 


Step 
# 


Number of 
Respondents 


Very 
Pertinent 


Not 

Pertinent Pertinent 


No 


Yes, 

Informally 


Yes, 

formally 


1 


5 


5 


0 0 


3 


2 


0 


2 


3 


2 


0 1 


2 


0 


1 


3 


3 


3 


0 0 


2 




0 


4 


1 


1 


0 0 


0 




0 


5 


2 


2 


0 0 


1 




0 


6 


2 


2 


0 0 


0 




1 


7 


2 


2 


0 0 


0 




1 



The pilot test Involved only a few schools (studied more or less case- 
study fashion), with a sample size that was not merely small; It was minlscule. 
However on several key questions regarding the KIT'S potential as an information 
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system, results tended to support the contention that the KIT gives users 
Information they view as Important, but that they rarely collect In a 
formal way. 

As can be seen with the Steps Involved In planning the evaluation 
(see Table 1), the Steps were overwhelmingly viewed as "Very Pertinent." 
However, evaluation-planning activities required by the Steps were rarely 
done with regard for a new program, or were done In an Informal way only. 
It appears that the KIT has the potential for giving people a framework for 
routine, formal gathering of pertinent Information. 

A national field-test sample of about 25 participating schools began 
using the KIT In the 1974-1975 school year. Results, although not yet 
complete, are consistent with the small-scale pilot test — the KIT has been 
perceived as being pertinent and as providing information not usually 
gathered with a new program. Furthermore, field-test sites have involved 
programs in many areas of the curriculum. Several of these programs are 
listed below: 

Program Name Location of School 

Continuous Progress Program Pennsylvania 
(cognitive objectives) 

Fresno Prime Reading Program California 

Self-Concept Development Georgia 

Psychomotor Skills Minnesota 

Basic Reading Skills Illinois 

Elementary Reading Program California 

Open Court Reading Program Arizona 

Plant Ecology Texas 

Field-test data on the use of the KIT are contained in the Appendix. 
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Table 2 

Responses of Field Test Users to the Language. 
Flexibility and Organization of the 
CSE Formative Evaluation KIT 





How do you rate the language and 
vocabulary of each part of the KIT? 


Step # 


Very Clear 


Clear 


Unclear 


Very Unclear 


1 


9 


8 


1 


0 


2 


9 


8 


1 


0 


3 


9 


8 


1 


0 


4 


9 


7 


2 


0 




How adaptable was the material to your own 
needs? Rate each section on Flexibility 




Verv Good 


Good 


Poor 


Very Poor 


1 
1 


7 


9 


1 


1 


2 


7 


6 


3 


2 


3 


7 


8 


2 


1 


4 


8 


5 


3 


2 




How well put together were the sections? 
Rate each part of the KIT on Organization 


Step # 


Very Good 


Good 


Poor 


Very Poor 


1 


9 


9 


0 


0 


2 


7 


8 


3 


0 


3 


9 


8 


1 


0 


4 


8 


8 


1 


1 



NOTE: Results based on questionnaires received prior to February 1, 1975, 
Field test continues to May 1975. 
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Table 3 

Responses of Fl^ld Test Users to Comparing 
the KIT to Previously Used Material 





Have you previously used material that was 
similar In purpose to parts of the KIT? 


OtcP If 




Yes 


No 


1 




6 


12 


C 




4 


14 


3 




5 


13 


4 




3 


15 




How does each part of the KIT compare 
with previously used material? 


Step # 


Very Good Good Poor 


Not 

Very Poor ApoTTcable 


1 


2 


4 0 


0 12 


2 


0 


3 1 


0 14 


3 


1 


4 0 


0 13 


4 


2 


1 0 


0 15 



NOTE: Results based on questionnaires received prior to February 1, 1975. 
Field test continues to May 1975. 
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